Marie-4: A High-Recall, Self-Improving Web Crawler That Finds Images Using Captions

page text describes associated images, and images are not captioned consistently. Content-based image retrieval systems that analyze the images themselves1 are progressing, but the systems require considerable image-preprocessing time. Furthermore, surveys of users doing image retrieval show that users are more interested in the identification of objects and actions depicted by images than in the color, shape, and other visual properties that most contentbased retrieval systems provide.2 Because object and action information is more easily obtained from captions, caption-based retrieval appears to be the only hope for broadly useful image retrieval.3 Commercial tools such as AltaVista’s Image Search search engine achieve respectable precision (the fraction of correct answers retrieved out of all answers retrieved) by indexing only “easy” pages, such as photograph libraries where images are one to a page and captions are easy to identify. Recall (the fraction of correct answers retrieved out of all correct answers) is equally or more important than precision, but users often do not realize how small it is for their queries. In experiments with 10 representative phrases, using pages retrieved by a traditional keyword-based Alta Vista search to calculate recall, we found Alta Vista’s Image Search had a precision of 0.46 and recall of 0.10. Higher recall requires dealing with a large variety of page layout formats and styles of captioning. Recent work has made important progress on general image indexing from the Web by intelligent information filtering of Web text.4–6 By looking for the right clues, large amounts of Web page text can be excluded as captions for any given image, and the captions in the remaining text can be inferred. Clues can include caption candidate wording, HTML constructs around the candidate, distance from the associated image, image-file name words, and associated image properties. These clues reduce the amount of text to examine to find captions, and the reduced text can be indexed and used for keyword-based retrieval. But so far, the selection of these clues has been intuitive, and there has been no careful study of the relative values of clues. This article reports on Marie-4 (see Figure 1), our latest in a series of caption-based image-retrieval systems.7 Marie-4 uses a wide range of clues, broader than any system we know about, to locate image-caption pairs in HTML Web pages. It is in part an expert system where the knowledge used is not especially novel in itself, but the synergy of a variety of knowledge working together provides surprisingly good performance. Unlike some caption-based retrieval systems3 and previous Marie systems, which require an image database with captions already extracted, Marie-4 is a Web crawler that autonomously searches the Web, locates captions using intelligent reasoning, and indexes them. It does not attempt full natural language processing and does not require the elaborate lexicon information of the earlier prototypes, so it is more flexible. Marie-4, a Web